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ABSTRACT 

This paper makes, the case that the intensifying pressure to 
assess higher educational outcomes objectively must be balanced by the 
inclusion of professional judgment. Mere numbers cannot tell the whole story 
of educational achievement. The tenacious attachment to positivism distances 
assessment and quality improvement from more inclusive ways of knowing. 
Assessment and quality improvement cannot avoid the knotty problem of how 
reality is discerned. The two key principles of positivism, the empirical 
reduction of all knowledge and the adequacy of mathematical analysis, cannot 
account for knowledge enacted through craftsmanship. It would be foolish to 
abandon rationally interpreted empirical data in assessment and quality 
improvement, but it is necessary to acknowledge and value intellectual 
judgment drawn from lived and reflective experience. Reflective practice is 
needed. The discernment needed to improve student learning does not lie 
primarily in finding ever more clever ways to measure students as determined 
organisms. The more productive approach lies in encouraging faculties to form 
communities of judgment to use the hard data and intuitive knowledge now 
available to them. (SLD) 
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Discerning Is More than Counting 

John Harris and Dennis Sansom 
Samford University 
March 2000 



The assessment movement, in pursuit of clarity and certainty, routinely rejects judgment in favor of 
objectivist knowledge. Consequently, assessment reduces learning to numerically measured and 
hierarchically ordered processes and outcomes. In some cases, such reductions make some kinds 
of knowledge manageable and definite. Yet not all aspects of learning are reducible to statistical 
categories. For all our efforts to reduce student learning to operational defmitions and precise 
numbers, the objectivist paradigm cannot capture all of it. 

If any human endeavors remain in our technologic society, surely higher education is one of them. 
And if substantive understanding is necessary for improvement that at least does no harm, can 
empirical scientific methods alone provide such understanding? Wilfi-ed McClay, in his article, 
“Clio’s Makeshift Laboratory,” said history cannot be done as pure science because it deals with 
human affairs. We believe this apphes to assessment and quality improvement in the most human 
of efforts, i.e. educating persons. 

The discipline of history is the science of incommensurable things and unrepeatable events. 
Which is to say, it is no science at all.. . .This melancholy truth may be a bitter pill to 
swallow, especially for those zealous modem sensibilities that crave precision more than 
they covet accuracy. But human affairs, by their very nature, cannot be made to conform 
to the scientific method — ^unless, that is, they are first divested of their humanness.' 

Thesis 

The intensifying pressure to assess higher educational outcomes obj ectively must be balanced by 
the inclusion of professional judgment. We do not take this position because higher education 
should not be accountable. Rather, objectivist data alone will not provide the substantive 
understanding needed to improve student learning and institutional performance. If the objectivist 
approach is pressed too far and too exclusively, it will do more harm than good. So we advocate 
making a prominent place for professional judgment in assessment for improvement and accountabihty. 
Both ancient wisdom and modem insight support the value of judgment as a complement to 
quantitative data. A number of eminent physical scientists makes the case for professional judgment 
that rests on intersubjective agreement, which, in turn, requires community. But lack of community 
reinforces the use of objectivist data; data not susceptible to local, subjective constmction or 
interpretation. There can be no place for professional judgment if the impersonal state attempts to 
hold educational institutions directly responsible for student performance without intermediary 
communities. These intermediary communities can be colleges, universities, or consortia of colleges 
and 



' Wilfred M. McClay, “Clio’s Makeshift Laboratory ,” Things (March 2001 , no. 1 1 1), 23. 



universities that develop internal or intramural ways of assessing requisite student learning for the 
degrees they award, reflecting on that learning, and enacting their reflections as improvements. 

Numbers Cannot Tell the Whole Story 

Michael Polanyi, one of the 20* century’s major critics of scientism and positivism, argues for the 
tacit foundation and dimensions of knowledge. He maintains that true intellectual contact with 
reality requires a “dwelling in and breaking out” experience.^ The “dwelling in” occurs when we 
submerse ourselves in an articulate fiamework* and begin to think about the experience according 
to the framework, not just to think about the framework. This submersion occurs in both arts and 
sciences, and Polanyi likened it to contemplation: “The task of inducing an intelligent contemplation 
of music and dramatic art [as well as science and mathematics] aims likewise at enabling a person 
to surrender himself to works of art. This is neither to observe nor to handle them, but to live in 
them.”’ 

In Seeing Like A State, James Scott connects our intense interest in numbers to the state’s efforts 
to imderstand and organize societies scientifically.'* Planned cities and forests demand “thin 
simplifications.” Yet these thin, formulaic, rationalistic approaches have led to ecological damage 
and unsatisfying human communities. “Any large social process or event will inevitably be far more 
complex than the schemata we can devise, prospectively or retrospectively, to map it.”’ He makes 
a case for the experience and judgment of a seasoned practitioner in complex and important 
situations over a theoretician without firsthand practical experience. “If your life depended on your 
ship coming through rough weather, you would surely prefer a successful captain with long experience 
to say, a brilliant physicist who had analyzed the natural laws of sailing but who had never sailed a 
vessel.”’ 

While Polanyi uses tacit knowledge, Scott employs the Greek word, metis, for intuitive judgment 
(“the faculty of advising, wisdom, coimsel, cunning, craft”’) derived from experience. “[T\echne, 
or art, is characterized by impersonal, often quantitative precision and a concern with explanation 
and verification, whereas metis is concerned with personal skill, or ‘touch, ’ and practical results.”* 

When contemplative knowing is thorough in its application and focused acutely on its object, one 
can experience a “breaking out” from a pre-established mode of interpretation to a direct experience 
of the object’s content, its ontological structure. This content cannot be adequately 



*A community of learners devises a language, or articulate framework, to share their experiences within the 
focus of their learning. 

^Michael Polanyi, Personal Knowledge: Towards a Post-Critical Philosophy (Chicago: University of 
Chicago Press, 1957), 195-202. 

Hbid., 196. 

Uames C. Scott, Seeing Like a State (Newhaven, CT: Yale University Press, 1998). 

Hbid., 309. 

Hbid.,Z\A. 

^ An Intermediate Greek-English Lexicon founded upon the Seventh Edition of Liddell and Scott s Greek- 
English Lexicon (Oxford: Oxford University Press, 1 989), 511. 

® Scott, 0/7. cit., 320. 
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and fully expressed either numerically or through classification. It is more truthfully expressed and 
rendered in terms of our appreciation and wonder of the object.* 

In stark contrast, the higher education assessment initiative and the later quality movement trace 
their methods chiefly to logical positivism, and the objectivist model of knowledge derived firom it. 
Edward L. Thorndike, a chief architect of educational measurement theory and practice, expresses 
educational assessment’s debt to positivism this way: 

If a thing exists, it exists in some amount. 

If it exists in some amount, it can be measured.’ 

Thorndike was a modem man, showing the influence of scientific objectivism in his thinking. Scott 
makes clear that modem institutions made their way by putting down historic ways of knowing. 

. . .a certain understanding of science, modernity, and development has so successfully 
structured the dominant discourse that all other kinds of knowledge are regarded as 
backward, static traditions, as old wives’ tales and superstitions. High modernism has 
needed this ‘ other, ’ this dark twin, in order to rhetorically present itself as the antidote to 
backwardness. The binary opposition also comes firom a history of competition between 
the institutions and personnel that sprang up around these two forms of knowledge. Modem 
research institutions, agricultural experiment stations, sellers of fertilizer and machinery, 
high-modemist city planners, third- world developers, and World Bank officials have, to a 
considerable degree, made their successful institutional way in the world by the systematic 
denigration of the practical knowledge that we call metis. 

Peter Ewell, perhaps the leading theorist and consultant on assessment in higher education today, 
links current fascination with numbers to the positivist-objectivist ethos of the 




*At this level, there is an aesthetic and moral dimension to knowledge. It is aesthetic because it is a form of 
appreciation of the object and of the accompanying intellectual and emotional satisfaction. It is moral because 
it is a faithful and honest response to the inherent features of the object. 

^Lee J. Cronbach, Essentials of Psychological Testing (New York: Harper & Row, 1 949). 

Cronbach’s wording of Edward L, Thorndike’s Credo is the wording commonly attributed to Thorndike. In trying to find 
this statement in Thorndike’s own work, I found the following sentences which convey the same meaning as Cronbach’s 
citation but in different words: 

Whatever exists at all exists in some amount. To know it thoroughly involves knowing its quantity as well as its 
quality. Education is concerned with changes in human beings; a change is a difference between two conditions; 
each of these conditions is known to us only by the products produced by it — things made, words spoken, acts 
performed, and the like. To measure any of these products means to define its amount in some way so that 
competent persons will know how large it is, better than they would without measurement. To measure a product 
well means so to define it in an amount that competent persons will know how large it is, with some precision, and 
that this knowledge may be conveniently recorded and used. This is the general Credo of those who, in the last 
decade, have been busy trying to extend and improve measurement of educational products. 

At the end of this passage, Thorndike added the following footnote: 

The conditions to be thoroughly known must be know as quantities a, b, c, d, etc., of qualities or powers, or skills, 
or knowledge A. B, Y, J. etc, — that is, an equation aA-^bB-^cY, etc. 

This direct quote from Thorndike comes from: Edward L. Thorndike, “The Nature, Purposes, and General Methods of 
Measurements of Educational Products,” Chapter II, The Seventh Yearbook of the National Society for the Study of 
Education. Part JJ The Measurement of Educational Products, Blooming, IL: Public School Publishing Company, 1918, 
16. 

Scott, op, city 1331-332. (The Greek word metis will be discussed later.) 
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Enlightenment. He cites the testimony of Joseph Addison, writing in The Spectator 200 years ago, 
who observes that “numbers are so much the measiu’e of everything that is valuable, that it is not 
possible to demonstrate the success of any action, or the prudence of any imdertaking without 
them.”" In Ewell’s view, the positivist tradition is, above all, “management by the numbers.”'^ 

Our tenacious attachment to positivism increasingly distances assessment and quality improvement 
from more inclusive ways of knowing. Furthermore, it separates assessment and quality improvement 
practitioners from W. Edwards Doming, to whom we are more indebted than any other quality 
theorist. Doming warns that reliance on numbers alone will destroy an organization: 

One cannot be successful on visible figures alone. Now of course, visible figures are 
important. There is payroll to meet, vendors to pay, taxes to pay; amortization, pension 
funds, and contingency funds to meet. But he that would run his company on visible 
figures alone will in time have neither company nor figures.'^ [Emphasis added.] 

Einstein is reported to have had the following quote on his Princeton office wall: “Not everything 
that coimts can be coimted, and not everything that can be coimted coimts.”'*' In our zeal to be 
quantitative and “scientific,” many assessment and quality improvement techniques would have 
tested Einstein’s composure: “I have little patience for scientists who take a board of wood, look 
for its thinnest part, and drill a great number of holes where the drilling is easy.”'^ 

We have to find our way between the Charybdis of knowing only through mmibers and technique, 
and the Scylla of knowing only by sensing. When McClay describes the two fallacies of knowing, 
we believe the same is true of higher education assessment and improvement: 

[T]he Fallacy of Misplaced Precision and the Fallacy of Misplaced Skepticism, are the 
extremes we want to avoid. There is a world of difference between saying that there is no 
truth and saying that no one is fully in possession of it. Yes, the truth is elusive, and only 
fleetingly and partially glimpsed outside the mind of God.'* 

As Ewell suggests, the assessment movement’s over-reliance on quantifiable measures exacts a 
price: “A maj or cost of rationalism, however, is its tendency to eliminate judgment — ^to allow the 
numbers alone to make decisions.”'’ Ewell adds that this overemphasis on numerical measurement 



" Peter T. Ewell, ‘“Heart and Minds’: Some Reflections on the Ideologies of Assessment.” AAHE Assessment 
Forum, Fourth National Conference on Assessment in Higher Education, Atlanta, June 21-24, 1989, 9. 

'^ibid: 10 . 

W. Edwards Deming, Out of the Crisis [MIT, 1 989), 1 2 1 . 

'‘‘Alice Calaprice, ed.. The Expanded Quotable Einstein (Newhaven, CT: Princeton University Press, 2000), 
318. (This quote appears in a section of quotes attributed to Einstein but not verified) 

Alice Calaprice, ed.. The Quotable Einstein. (Newhaven, CT: The Hebrew University of Jerusalem and 
Princeton University Press, 1996), 185. 

'* McClay, op. cit., 27. 
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drives people “to move the numbers in the desired direction, and not necessarily to undertake real 
improvements.”'* 

The 1 T"* century mathematician and philosopher Pascal observes in his Pensees that “mankind 
suffers from two excesses: to exclude reason and to live by nothing but reason.”'^ Unless we in the 
assessment community leam to steer a middle course, we risk being wrecked by one or the other 
of these excesses. Pascal goes on to identify two ways of thinking associated with these extremes — 
the mathematical and intuitive: 

In the one [i.e., the mathematical] the principles are palpable, but removed from ordinary 
use; so that for want of habit it is difficult to turn one’s mind in that direction. . . .But in the 
intuitive mind the principles are found in common use, and are before the eyes of everybody. 
One has only to look, and no effort is necessary; it is only a question of good eyesight.^ 

Both mathematical and intuitive minds have assets and liabilities. Assessment and quality 
improvement experts generally emphasize problems associated with the intuitive mind. We find 
fault with its workings because: 1) it evades precise definitions; 2) its results are not easily and 
clearly measurable; and 3) its processes are not predictable and controllable. Trying so hard to 
avoid intuition’s Scylla, we are pulled under by Charybdis, the mathematical mind. Reflecting on 
Pascal’s thought, Jacques Barzun describes the underlying problems of scientism: 

Scientism is the fallacy of believing that the method of science must be used on all forms of 
experience and, given time, will settle every issue. Again and again, the bright thought has 
occurred, “If we can only define our terms, if we can then measure and reason flawlessly, 
we shall have created one more science.” 

Barzun concludes: 

The clue to the fallacy of SCIENTISM is this: geometry (in all senses of the term [i.e. all 
mathematics]) is an ABSTRACTION from experience; it could not exist without the work 
of the human mind on what it encounters in the world. Hence the realm of abstraction, 
useful and far from unreal, is thin and bare and poorer than the world it is drawn from. 
It is therefore an idle dream to think of someday getting along without direct dealings 
with what abstraction leaves untouched. The meaning of this contrast is that the 
enterprise of science has its limits. [Emphasis added.] 

Deming rejects simplistic tools of quantification ironically misused often by quahty improvement 
specialists. He may have learned to mistrust their excessive reliance on 



’’Ewell, op. cit., 12. 

'* ibid. 

'Ubid.. 135. 

’"Blaise Pascal, The Thoughts of Blaise Pascal [Translation of Pensees] (Westport, CN: Greenwood Press, 
1 978, Reprint of the 1 96 1 ed. published by Dolphin Books, Garden City, N. Y.), 7-8. 

” Jacques Barzun, From Dawn to Decadence (New York; Harper Collins Publishers, 2000), 218. 
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numerical measurement through close reading of C.I. Lewis’ Mind and the World Order, a work 
he often cites approvingly. According to Lewis, no matter how strongly we believe in the adequacy 
of numerical measurement, “we must always bear in mind that empirical evidence is never 
complete.”^^ In contrast to many of his disciples, Deming recognizes the epistemological limitations 
of exclusive rationalism and empiricism as do mathematicians and physical scientists. 

Epistemology Is Not Just Academic 

Assessment and quality improvement cannot avoid the knotty problems of how we know, of how 
we discern, reality. Lewis knows reality ca nn ot be discerned exclusively by tacit or objectivist 
knowing. We must accept a kind of double-tmth: there are certainties, such as those of mathematics, 
which directly concern only what is abstract; and there are presentations of our sense-experience 
to which we seek to apply them, but with a resultant empirical truth which may be no more than 
probable.^ 

Lewis believes the quantifiable order we report seeing is an order we necessarily impose on our 
experience of the world. Many experiences lack the imiformity required for prediction; and if they 
exhibited such imiformity, we could do nothing to change them: “It may seem that [uniformity] is 
precisely what is required for knowledge and prediction. But it is not: in a world so constituted, 
whatever could be learned would not be worth knowing, because nothing could be done about 

it»24 

We must learn to question overly precise planning schemes and organizational change models, 
especially those engineered on relatively imreliable psychosocial measures that consistently yield 
correlations far weaker than those in more exact physical sciences. For example, a major study of 
physician academic and practice performance showed essentially no relation among 80 factors.” 
This study involved 1 02 medical faculty at the University of Utah, 190 urban specialists practicing 
in the Ogden-Salt Lake-Provo area, and 217 urban and rural general practitioners. The researchers 
reached the following conclusions : 

1. Medical School Grades are Inadequate as Guidance or Predictive Tool for Later Physician 
Performance: 

According to the evidence in this study the presently available grades are generally of 
little or no use in identifying among the graduating M.D. ’s those who well be more 
successful in any of the 25 or more obtained dimensions of physician 
performance. . . Regardless of any possible degree of restriction of range of talent. 
present academic grades on graduating medical students simply do not predict how 
well a medical student will perform in medical practice. 



” Clarence Irving Lewis, World Order (H&n York: Dover Publications, Inc., 1929), 133 

^^ibid., ix. 

^Hbid., 355. 

” Philip B., Taylor, Calvin W., Richards, and James M. Jr. Price, and Tony Jacobsen, Performance Measures 
of Physicians (University of Utah: The research reported herein was supported through Cooperative 
Research Program of the Office of Education, U. S. Department of Health, Education, and Welfare, Contract 
No. OE-2-10-093), August, 1663. 
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2. Medical School Grades are Inadequate as Substitute Criteria for On-The-Job Performance 
ofPhysicians: 

Our fin din gs (without any attempts at corrections) consistently and clearly demonstrate 
that performance in a formal academic setting, as measured by grade-point-averages, 
is almost completely independent of all measured performances as a practicing 
physician. . .the highest single correlation obtained between academic grades and our 
performance measures was only +.30. Even with extremely generous^corrections for 
re.striction of range of talent we further believe that it is highly unlikely that more than 
one or two physician performance factors could be approximated by academic grades.^ 

A second, very consistent finding of this research was that academic achievement 
does not bear a positive relationship to performance as a practicing researcher, 
academician, or physician.^’ (The underlining for emphasis is that of the original authors.) 



This study dramatically illustrates general findings that grades may correlate with other grades but 
with few other variables. And when they do, the correlations are too low to predict anything with 
much certainty for an individual. 

The low or no correlation between grades and post-graduation performance is indicative of other 
measures of human performance. We do not have an empirical science of human behavior with 
predictive capabilities capable of supporting an impersonal technology for designing, managing, 
and reforming human organization. 

Beyond the unreliability of almost all human measurements, there is another problem of the multiplicity 
of variables affecting individual and collective behavior. Who can know all the variables involved, 
much less measure them all reliably? Here we come to a consistent criticism of assessment in 
higher education: It emphasizes what it can measure, ignoring those factors not susceptible to 
empirical measurement. Often, academic colleagues see assessment as majoring in minors. 

We, like the Greeks, seek techne as eliminating tuche, chance. Nussbaum points out that techne 
is more likely to eliminate chance when there is a single end rather than multiple ends: “[A]n end 
which permits of quantitative measure seems to yield more precision than an end that cannot be 
so measured.”^* 

[I]t is possible to have a bona fide techne that will be qualitative, plural in its ends, and in 
which the art activities themselves constitute the end, but such a techne seems unlikely to 
yield the precision and control that would be yielded by an art with a single, quantitatively 



^Ubid.,\Q6. 

^Ubid.,\\\. 

Martha C. Nussbaum, The Fragility of Goodness: Luck and Ethics in Greek Tragedy and Philosophy. 
(Cambridge: Cambridge University Press, 1986), 99. 



ERJC 



13 



measurable, external goal.^’ 



Beyond Na ve Rationalism 

Lewis and Polanyi both address epistemological limits of scientism based on the illusion of empirical 
predictability and control of experience. Lewis, as a pragmatist philosopher, believes that our 
sense of order in nature is experienced in generalities. “Reality is more orderly than experience, 
because reality is experience categorized.”^" It is not necessary to posit absolute uniformity in 
nature to make predictions through quantifiable models. Yet given the lack of total uniformity of 
experience, Lewis suggested that we think in terms of probabilities rather than certainties. 

Though trained as a chemist, Polanyi increasingly turns his attention to philosophical presuppositions 
underlying the unquestioned objectivity of science. He wrote Personal Knowledge “to show that 
complete objectivity as usually attributed to the exact sciences is a delusion and is in fact a false 
ideal.”^' Polanyi believes scientists are guided by tacit knowledge, not unlike artisans. Master 
craftsmen frequently cannot explain to beginners how they achieve results in explicit, rational, 
ordered statements. Polanyi writes: “It is pathetic to watch the endless efforts — equipped with 
microscopy and chemistry, with mathematics and electronics — ^to reproduce a single violin of the 
kind the half-literate Stradivarius turned out as a matter of routine more than 200 years ago.”“ 

Two key principles of positivism — the empirical reduction of all knowledge and the adequacy of 
mathematical analysis — cannot account for knowledge enacted through craftsmanship. On the 
one hand, empirical positivism may have tools to explain limited areas of knowledge (e.g., simple 
domains of sense experiences). On the other, when it enters fields involving judgment, skill or 
connoisseurship, it is forced to reduce them to the lowest common denominators of sense experience 
and mathematical analysis. Much of what we do in education cannot be made to fit positivism’s 
Procrustean bed. 

For Polanyi, such reduction is wrong because it does not refiect realities of normal human action. 
We could not manage most of our personal and even much of our professional lives if we were 
restricted to living only by empirically supported research and mathematical formulas. Even the 
most rigorous exact scientists, Polanyi observes, depend on tacit knowledge in their work: “We 
shall find personal knowledge manifested in the appreciation of probability and of order in the 
exact sciences, and see it at work even more extensively in the way the descriptive sciences rely on 
skills and connoisseiu^hip.”” 



The Limits of Language 

No matter how rigorously we insist on empirically proven, rationally articulated and quantified 
knowledge, we must in the end admit that we cannot articulate all we know. On one hand. 



ibid. 

Lewis, op. city 365. 
^‘Polanyi, op. city 18. 

ibid., 53. 

^^ ibid ., 17. 
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language is a distinguishing ability of humankind. We cannot imagine thinking or reasoning without 
verbal and quantitative symbols or sounds. On the other, as the physicist and philosopher Percy 
Bridgman writes, “No linguistic stmcture is enable ofreproducing the full complexity of reahty. . 

Knowledge requires concepts, and we mostly think with words. One’s language comes from immersion 
in a language co mmuni ty Language does not follow rigid rules, and sentence formation does not 
evolve in exact ways. A speaker does not have to know all the possibilities of a language to speak 
meaningfully and conceive viable concepts. Nevertheless, we must know the creative principles of 
the language’s semantics and syntax. Each time we make a knowledge claim, we tacitly rely on the 
complex functioning of the language community. 

This tacit dimension of knowledge raises the point of the moral accountability of knowledge. The 
objectivist approach tries to safeguard knowledge from misinformation and misuse by basing it on 
a sure and certain methodology. But no methodology can entirely compensate for the social and 
imprecise way knowledge is gained and used in a language community. Because knowledge is 
gained through give and take, continual revising, and inexactitude of social existence, tmth statements 
presuppose the trustworthiness of the knower, as well as the knower’s ability, to describe, analyze, 
interpret and articulate knowledge. Polanyi speaks of an intellectual vocation: “Believing as I do in 
thejustification of deliberate intellectual commitments, I accept these accidents of personal existence 
as the concrete opportunities for exercising our personal responsibility. This acceptance is the 
sense of my calling.”^^ 

The intellectual’s calling in society is to form intellectual commitments to ontologically superior 
reahties within the accidental features of social experience, such as that which causes moral courage, 
aesthetic responses and spiritual insights. Our knowledge of the world is not just an intramural 
dialogue of words to words, but of words serving a moral purpose to express and display the 
content of what the mind can contemplate. Perhaps the greatest failure of the objectivist approach 
occurs here — it implicitly absolves us of being personally responsible for our knowledge. 

PowerPoint Fictions 

Techniques of rhetoric have often tempted speakers to shave the facts for the sake of effect. 
Whether we consider Marc Anthony’s “Friends, Romans, Countrymen” speech or a quality 
improvement consultant’s PowerPoint presentation, some truth is usually sacrificed to make the 
sale. The rhetorical power of direct and succinct language often leads speakers to overgeneralize, 
oversimplify, and stretch the facts. After hearing a Bible story, a parishioner said, “Preacher, I 
don’t remember that story the way you told it.” The preacher replied, “If that’s not the way it is, 
that’s the way it ought to be.” 

This emphasis on effectiveness at the expense of tmth is endlessly repeated on the quahty improvement 
conference circuit. Numerous presenters display PowerPoint diagrams. 



Cited in Robert Greenleaf, The Servant as Leader [written in 1 970] (Indianapolis: The Robert K. Greenleaf 
Center, 1991), 11. 

^’Polanyi, op. cit., 322. 
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explaining how they achieved clear-cut improvements by their design, rollout, and implementation of 
a rationally-designed improvement plan. We are often expected to believe the presenter charted 
every step of the improvement and implemented it without a hitch. When correlation coefficients 
between dependent and independent variables in the social sciences, such as correlations between 
aptitude and achievement measures, rarely, if ever, exceed . 60 , and when human typical performance 
correlations, such as correlations of personality inventories and observed behavior, are even lower, 
we should suspect that more order was projected into the realities involved than from which it was 
drawn. (See Furfey’s and Daly’s explanation of why social science measures correlate so much 
lower than physical science measures.*) Such diagrams cannot accurately represent the disorderly 
messes inevitably associated with human organizations. First, what is actually involved in organizational 
change cannot be known. And if it could be, it would either be very difficult to draw, or the complexity 
of doing so would defeat the effort to increase understanding. Furthermore, many, if not most, 
changes in organizations involve personahties and political issues that cannot be discussed pubhcly. 
We in assessment should heed McClay’s warning to historians. 

History strives, like all serious thought, for the clarity of abstraction. We would like to 
make its insights as pure as geometry. . . .But its subject matter — ^the tangled lives of human 
beings, in their unique capacity to be both subject and object, cause and effect, active and 
passive, free and situated — forces us to rule out that goal in advance.^ 

Obviously, assessment and quality improvement draw upon social sciences, i.e. psychology, 
sociology, political science. These sciences rarely have precise measures and highly predictive 
accuracy, the hallmarks of science. Scott remarks on how social sciences have mimicked the 
techne of hard sciences, attempting to remove chance. 

A recurrent theme of Western philosophy and science, including social science, has been 
the attempt to reformulate systems of knowledge to bracket uncertainty and thereby permit 
the kind of logical deductive rigor possessed by Euclidean geometry.” 

His endnote for the above statement expresses our concern about the present state of educational 
assessment, which is so dependent on the social science paradigm: 

Borrowing the prestige of scientific language and methods from the biological sciences, 
many social scientists have envisioned and tried to effect an objective, precise, and strictly 
replicable set of techniques — a set of techniques that gives impartial and quantitative 
answers.. . .TTiey achieve impartiality, precision, and replicability at the cost of 

*The physicist, in contrast to the psychologist, deals with ‘‘functional association” in which “one and only one 
value of one variable is associated with each value of the other.” In contrast, the psychologist deals with the 
situation where “a set of different values of one variable is associated with each value of the other.” Furfey and 
Daly give the example that if children are tested for their intelligence and reading ability, children with the same 
intelligence scores will have different reading scores. They point out that despite the value of tests for intelli- 
gence and achievement, their errors of measurement can be expressed in percentage points, while physical 
science measures are ”where an accuracy of one part in a hundred thousand is quite frequently attained, while 
under specially favorable conditions the error is reduced to a small fraction of this insignificant figure.” Paul 
Hanly Furfey and Joseph F. Daly. “The Interpretation of the Product-Moment Correlation CoefHcient.” The 
Catholic University of America Educational Research Monographs, Thomas G. Foran, Editor, vol. Viii, No. 4, 
June 1, 1934, pp. 1, 4. 

^^McClay, op. cit.^ 24. 

Scott, op. czV., 321. 
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accuracy [Emphasis added] 

Beyond all the epistemological limitations of getting to the reality of anything, there are moral 
dimensions of data collection and reporting. It is well known that responses to opinion polls and 
satisfaction surveys can be greatly influenced by the way questions are flamed and the respondents’ 
predispositions. We can cherry-pick what we report in many ways for all kinds of reasons. The 
statisticians’ in-house joke about torturing data until it confesses can be pathetically true for the 
unscmpulous or overzealous analyst. Statisticians know all too well that two analysts drawing on 
the same data pool with diflerent motivations can manipulate data to yield very diflerent interpretations 
and impressions. The more we know about statistics, the more we want to know about them 
before accepting them. JimNetherton, a mathematician and former Samford Provost, demonstrated 
this time and again. He would consistently ask how the data were collected and how numbers 
were assigned to various properties before becoming interested in the statistical formulas used to 
manipulate the data. 

Neil Postman begins Technopoly, his critique of technique-dominated society, with Socrates’ 
account of Thamus’ criticism of Theuth’s invention of writing. Thamus says to Theuth: 

What you have discovered is a recipe for recollection, not for memory. As for wisdom, 
your pupils will have the reputation for it without the reality: they will receive a quantity of 
information without proper instruction, and in consequence be thought very knowledgeable 
when they are for the most part quite ignorant. And because they are filled with the conceit 
of wisdom instead of real wisdom they will be a burden to society.^’ 

If writing on papyrus or parchment with a sharpened stick or quill dipped in some colored liquid 
deluded humankind into unwittingly substituting information for understanding, what would Thamus 
say about high-speed printing presses, telephones, movies, TV, computers, and the Internet? 
Furthermore, Postman traces the practice of assigning numerical weights to students’ thoughts to a 
Cambridge tutor, William Parish, in 1 792. While we know little beyond this about Parish, Postman 
believes “his idea that a quantitative value should be assigned to human thoughts was a major step 
toward constructing a mathematical concept of reality.” Postman then adds: 

If a number can be given to the quality of a thought, then a number can be given to the 
qualities of mercy, love, hate, beauty, creativity, intelligence, even sanity itself When Galileo 
said that the language of nature is written in mathematics, he did not mean to include human 
feeling, accomplishment, or insight. But most of us are now inclined to make these inclusions. 
Our psychologists, sociologists, and educators find it quite impossible to do their work 
without numbers. They believe without numbers they cannot acquire or express 
authentic knowledge.*'^ [Emphasis added.] 



^Hbid., 426. 

^’Neil Postman, Technopoly: The Surrender of Culture to Technology (New York: Vintage Books), 4. 



Reinstating Tacit Knowledge 



It would be foolish to abandon rationally interpreted empirical data in assessment and quality 
improvement, but we do need to renew the license of tacit knowledge and ineffable wisdom. This 
does not mean turning to astrology, Ouija boards, or seances. It means acknowledging and valuing 
intellectual judgments drawn from lived and reflective experience that may not necessarily be 
explicitly and completely explained in an orderly diagram or logically organized paper. 

PowerPoint-like presentations can and often do oversimplify human affairs and emotions into 
numerical values and mechanical diagrams that do not actually express the event. More importantly, 
they shape the content we select to communicate. Postman puts it this way: “[Ejmbedded in every 
tool is an ideological bias, a predisposition to constmct the world as one thing rather than another, 
to value one thin g over another, to amplify one sense or skill or attitude more loudly than another.”'" 
He goes on to cite Wittengenstein’s point “that language is not only the vehicle of thought but also 
the driver;” as he suggests, this is McLuhan’s point, “The medium is the message.”''^ Citing the old 
adage that to man with a hammer everything looks like a nail. Postman apphes it to commumcation 
media as follows: 

To a man with a pencil, everything looks like a list. To a man with a camera, everything 
looks like an image. To a man with a computer, everything looks like data. And to a man 
with a grade book sheet, everything looks like a number.''^ 

As Postman makes clear, new tools and techniques are invented in good faith but often result in 
xmintended outcomes. For example, he cites the invention of the clock by Benedictine monks in 
the twelfth and thirteenth centuries. They developed the clock to be more punctual in their daily 
rituals. Yet, the clock, as he points out, brought a “precise regularity;” which in turn as Mumford, 
“The mechanical clock made possible the idea of regular production, regular working hours and a 
standardized product.”'^ 

So the clock intended to increase devotion to God made capitalism possible. So we have the 
paradox of the invention having reverse of its intended effect. 

The paradox, the surprise, and the wonder are that the clock was invented by who wanted 
to devote themselves more rigorously to God; it ended as the technology of greatest use to 
men who wished to devote themselves to the accumulation of money. In the eternal stmggle 
between God and Mammon, the clock quite unpredictably favored the latter."’ 

It is interesting that Michael Power, professor of accounting and finance at the London School of 
Economics and Political Science, suggests that auditing may actually lead to mediocrity. 



*^ibid., 11 . 
ibid., 13 . 
ibid, 13 - 14 . 
^Ubid., 14 . 

"" ibid, 14-15 
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[T]he evaluation of teachers and schools by exam results creates safe teaching without 
explorations “off-syllabus.” In short, audits with quality as their objective may lead to 
unintended forms of mediocrity as games of “creative comphance” develop around auditable 
information systems.'^ 

So as new technologies occur we should be aware that they are not neutral or objective. They not 
only shape how, but also what, we communicate while deluding us with the notion that we know 
more with them than before. There is a “conceit of wisdom” to such displays because their 
apparent preciseness conveys more order than what exists; and coupled with elaborate statistical 
analyses, they imply certainty and predictability beyond the capabilities of the information. 

Working with an experienced executive trying to choose between two individuals for an important 
position, a consultant said, “I’m going to flip a quarter. Heads it’s Bill and tails it’s Jim.” After 
flipping the coin and immediately covering it, the consultant asked, “Tell me quick, do you want it 
to be heads or tails?” This exercise helped surface the executive’s gut sense, while trying to work 
through a rational set of pluses and min uses had left the executive unsure. The underlying assumption 
is that the subconscious mind remembers more than the conscious one, and that a professional 
with years of experience has a deep well of knowledge that cannot always be reached through a 
calculus of benefits and liabilities. Scott graphically describes metis at work in stories about a 
physician’s uncanny ability to detect syphilis at a very early stage, and an Indonesian sea captain’s 
abilities to sense slight sailing changes while fast asleep below deck. The practitioner learns to 
attend to very slight cues that may not break into consciousness, but that may be detected by a 
keen observer.* 

Polanyi describes tacit knowledge: 

When I speak of ineffable knowledge, this should be taken literally and not as a designation 
of mystic experience, to which I do not wish to refer at this stage. Even so my attempt to 
speak of the ineffable maybe thought to be logically meaningless, or alternatively, to offend 
against the Cartesian doctrine of “clear and distinct ideas” which the early Wittgenstein 
transposed into terms of semantics in his aphorism: “Of what cannot be said” — i.e. said 
exactly, as a sentence in natural science — “therefore one must 

*Scott explains: Metis knowledge is often so implicit and automatic that its bearer is at loss to explain it... a physician 
who, at the turn of the century (the beginning of the 20*^) had a spectacularly high success rate in diagnosing syphilis 
in its early stages. Laboratory tests confirmed his diagnoses, but he himself did not know precisely what it was that 
he detected in the physical exams that led him to his conclusions. Intrigued by his success, hospital administrators 
asked two other doctors to closely observe his examination of patients over several weeks to see if they could spot 
what he was picking up. At long last, they and the doctor realized that he was unconsciously registering the patients’ 
slight eye tremor. The eye tremor became a universally recognized symptom of syphilis. Although this insight could 
be codiHed, what is instructive here is that it could have been achieved only through close observation and long 
clinical experience and that, even before then, it could have been known subliminally. 

Any experienced practitioner of a skill or craft will develop a large repertoire of moves, visual judgments, a 
sense of touch, or a discriminating Gestalt for assessing the work as well as a range of accurate intuitions born 
of experience that defy being communicated apart from practice. Scott illustrates this by stories such as an 
Indonesian sea captain who would awaken immediately to any slight change in direction, weather, or current; a 
doctor who could detect diphtheria by odor when first stepping into a house; or a farmer who can tell the health 
of soil merely by looking at it. Scott, op cit, p. 329. 

15 , 

Michael Power, ‘The Perils of the Audit Society,” George Jones and Steve John, eds. (LSE Public Service 
Seminar Series: Government Department, London School of Economics and Political Science, 1 997), 8. 
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be silent”. . .These observations [referring to extensive examples given elsewhere in the 
book] show that strictly speaking nothing that we know can be said precisely;’ and so 
what I call “ineffable” may simply mean something that I know and can describe even less 
precisely than usual, or even only very vaguely.^’ 

The reference to Wittgenstein is quite revealing. Wittgenstein maintains in Tractus Logico- 
Philosophicus, from which the above quote comes, that what can be said, can be said clearly by 
natural science and logic. Nevertheless, he also admits that clear knowledge has its hmits. It cannot 
express the bases of moral and aesthetic knowledge, which he calls mystical. Poets and religious 
behevers experience such knowledge. The proper response to the mystical and ineffable is a moral 
and aesthetically motivated hfe, not necessarily a logical and scientific exactness. 

Greenleaf, in The Servant as Leader (1970), writes that “there is usually an information gap 
between the solid information in hand and what is needed.”“* He follows this with: 

Every once in a while a leader finds himself needing to think like a scientist, an artist, or a 
poet. And his thought processes may be just as fanciful as theirs — and as fallible. Intuition 
is a feel for patterns, the ability to generalize based on what has happened previously. The 
wise leader knows when to bet on these intuitive leads, but he always knows that he is 
betting on percentages — his hunches are not seen as eternal truths.“’ 

Statistical information contributes greatly to assessing human performance, curricular and 
instructional effectiveness, and process reliability and efficiency. But decisions require a tacit 
dimension in addition to data; a dimension not entirely amenable to flow charts, graphs, and 
PowerPoint demonstrations. 



Assessment and QI’s Future 

On a satellite telecast in the mid-90s, J. M. Juran, a Deming peer, was asked, “What is the greatest 
threat to the quality movement?” His answer was, “The quality professionals.” If we as specialists 
in quality assessment want a future, we have to broaden our ideas of how we discern reality. 
Assessment and quality improvement are inseparable. Assessment is about discerning the realities 
of student learning, curricular and instructional effectiveness, student satisfaction, and costs. Since 
quality improvement’s grasp of successful teaching and value formations cannot reach beyond its 
discernment, we must add tacit knowing to empirical analysis. We do not have to choose between 
objective and subjective knowing. Empirical data can reveal counterintuitive realities, but 
interpretation and evaluation of moral, aesthetic, and spiritual experiences require tacit knowledge. 



* At this point, Polanyi cites A. N. Whitehead in his Essays in Science and Philosophy, London, 1948, p. 73. “There 
is not a sentence which adequately states its own meaning. There is always a background of presupposition which 
defies analysis by reason of its infinitude.” 

^’Polanyi, op cit., 87-88. 

Greenleaf, op. cit., 15. 
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Nevertheless, conventional assessment strategies deliberately exclude the personal for the objective, 
the unique for the common, and the tacit for the explicit. To support and improve assessment and 
quality improvement abilities, we depend too much on general theories and techniques of behavior 
explanation and control in contrast to techniques relevant to specific disciplines and personal 
uniqueness. Samuel Hope puts it this way: 

In days when we understood diversity more comprehensively, there was acceptance of the 
fact that fields and responsibilities and functions could be different, that each could have its 
own particular mix of values, that each had different evaluation mechanisms appropriate to its 
nature and purposes. Expertise was not just a body of knowledge and skills, it was their use 
in pursuit of wisdom, judgment, and service. Massive belief in technique, and especially 
technique driven by theory, have led us away fi'om these reasonable positions. Vast energies 
have been poured into building up public feith in management technique, economic technique, 
assessment technique, political technique, public relations technique, and many others as 
transcendent powers that operate according to principles irrespective of purpose or 

COntent.^'^ [Emphasis added] 

In other words, Hope tells us, “One size does not fit all, and you assessment and quality technicians 
should stop stretching and cutting us to fit your procrustean beds!” In efforts to obtain institution or 
system-wide comparability, uniform assessment techniques and instruments are imposed, ignoring 
differences among disciplines in their modes of inquiry. Whether intended or not, there is a power 
game here. It’s not unlike the conqueror insisting that the conquered adopt the conqueror’s language 
and forget theirs. Later in his presentation, Hope claims: 

[T]he future of assessment and accountability depends on which set of values are at the 
core, the state of those values as determined by fashion, questions of technique, and what 
body of expertise is trusted with the task.... As leaders [in the arts], we will be countering 
many notions if we are to protect the essence and spirit of the arts on our campuses.’' 

On the other hand, assessment experts are perceived, in many cases justifiably, as trying to displace 
such discipline-fiiendly assessment. Hope provides evidence of this perception: 

In the assessment and accountability realm, we are being asked to believe that those with 
expertise in the disciplines are unqualified to assess themselves and to produce evidence 
of their own accountability. Accountability techniques based on mission and content — in 
our case, the mission and content of the arts — are to be superseded by those who are 
experts in assessment technique. It is in this condition that assessment and accountabihty 
can become the enemy of quality, the enemy of substance, the enemy of integrity, the 
enemy of wisdom-based judgment, and certainly the enemy of work in and with content.” 



“Samuel Hope, “Challenging Notions of Program Assessment” (Presented to the International Council of 
Fine Arts Deans, St. Louis, Nov. 5, 1998), 3. 

^'ibid., 4. 




15 



O 



The first task of assessment is not to impose one assessment template to yield comparable data; 
rather, it is to encourage faculties to develop assessment strategies and instruments consistent with 
the modes of inquiry associated with their disciplines. If improvement is the primary obj ective, then 
data comparability is not the first concern. If accountability is the primary driver, then comparability 
is a prime concern; but we believe it comes at the expense of improvement initiated within the 
respective faculties. 



Engineered Colleges? 

The promise of certainty and control of systems are irresistible in a society committed to achieving 
the greatest material benefits at the least cost. And as Jacques Ellul makes clear, all institutions and 
conventions of a technological society inevitably yield to efficiency; efficiency is as immutable in 
modem, materiahstic societies as the law of gravity.^^ Education is now more justified for its economic 
value rather than its liberal, moral benefits. Since instrumental knowledge is indispensable in modem 
economies, they will find the most efficient ways to disperse it. We should expect that most efforts 
to improve education will focus on instrumental knowledge and its efficient dispersion. 

If engineered systems based on highly predictive sciences have produced so many successful and 
cost-effective results in manufacturing, why should a technological society spend time and money 
developing idiosyncratic teachers? It is obviously more cost-effective to engineer instructional 
systems and to retain and train learning facilitators to mediate them. The next step is to engineer 
these smaller learning systems into seamless, institution-wide learning systems; that is, engineer the 
university as a person-independent megasystem of learning and engineer out the human uniquenesses 
and proclivities, and engineer in person-independent processes and integrated systems. This vision 
of a university differs dramatically from that of developing each person ’s uniqueness* And 
similar to Scott’s observations about rationally planned cities and scientifically managed forests, 
we believe rationally engineered institutions will prove to be uninteresting as geometric cities and as 
vulnerable as monocultural forests stripped of their diversity.^ 

We are interested in ways to improve continually human proportioned and variegated institutions 
focused on optimizing students as individuals through learning. We believe improvement lies in 
communities of practitioners reflecting on quantitative data and their tacit knowledge or metis. 
This does not exclude the engineered view, but in this scenario it serves rather than rules. 



‘Another view: Reality may be understood ultimately as personal and relational. Personhood is maximized 
through nurturing relationships. One is most personal when one’s joy comes from others’ joy. The individual 
never becomes a person by knowledge alone. Since loving others is not a natural state, it occurs only through a 
supra-natural relationship. Within this perspective, the role of education is to nurture persons through relation- 
ships with the Ultimate Person and others. While humankind is far superior to the highest animals in intelli- 
gence and language, the most distinguishing human characteristic is the potentiality of becoming a person in the 
image of God. Education that concentrates only on instrumental knowledge for individual and socioeconomic 
advantage may produce rational, autonomous individuals at the expense of their personhood. The current Zeit- 
geist emphasizes knowing for self-sufficiency; the contrary view emphasizes knowing the other for community. 

ibid. 

Jacques Ellul, The Technological Society (New York: Alfred A. Knopf, 1964). 
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Reflection: The Key to Improving Core Results 



Paul Batalden* suggests that broad improvement in any organization can begin effectively “by 
determining what’s imacceptable in its core business.” Most colleges and universities would agree 
that student learning is their core business. Few, if any, human imdertaking could be more complex 
than facihtating higher-order human learning. No other enterprise is more in need of Scott’s metis. 

Metis, with the premium it places on practical knowledge, experience, and stochastic 
reasoning, is of course not merely the now-superseded precursor of scientific knowledge. 
It is the mode of reasoning most appropriate to complex material and social tasks where 
the imcertainties are so daimting that we must trust our (experienced) intuition and feel our 
way.^’ 

We believe Schon’s’® strategy of reflective practitioners is one effective way faculty may engage 
their merits for improving student learning. This means encouraging faculty groups to form themselves 
into communities of judgment to leam to reflect on the interaction of their discipline’s content and 
structure with relevant learning and assessment theory and praxis. 

Samford University faculty involved in imdergraduate problem-based learning [PBL] are becoming 
reflective practitioners through writing course portfolios and having them peer-reviewed. Thirty- 
nine PBL course portfolios have been or are being peer-reviewed by a national content peer and 
a national instmctional design or PBL expert.” One yoimger faculty member said that he had been 
interested in good teaching and considered himself a good instructor, but writing his course portfolio 
and having it nationally peer-reviewed led him to see his teaching and his students’ learning as 
objects of scholarship. The scholarship ofteaching is one way the academy can implement Schon’s 
reflective practitioner ideas. 

Improvement Through Reflection 

We know of no formula for infusing “practitioner reflection” into a university, and it would obviously 
be inconsistent with our thesis to introduce one. At the same time, we offer four academic 
applications of Schon’s basic principles of reflective practice to facilitate faculties’ use of tacit 
knowledge or metis to improve student learning and institutional practices: 



* Paul Batalden, Professor and Director, Health Care Improvement, Leadership Development, Center for the 
Evaluative Clinical Sciences, Dartmouth Medical School, personal communication July 6, 2000. 

Scott, op. cit, 134-142, 19-20 

Scott, 0/7. cit.y321. 

Donald A. Schon, The Reflective Practitioner: How Professionals Think in Action (Basic Books, 1983). 
a recent meeting with the President and Acting Provost, faculty commented on their PBL experiences. Eric 
Fournier in Geography reported that among other benefits he had moved from being a good teacher to a 
scholarly one. Before PBL, he practiced good teaching techniques relatively well, and students rated him 
favorably. After receiving training in the systematic design of PBL, writing a course portfolio intentionally 
designed to promote reflection on the design, field testing, assessment, and improvement of PBL instruction, 
and receiving the comments of two national peer reviewers, he now reflects on his teaching praxis from a 
scholarly perspective and focuses more on student learning than teaching. 

O 
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Principle 1 : ‘ ‘Let us search ... for an epistemology of practice implicit in the artistic, intuitive 
processes, which some practitioners do bring to situations of uncertainty, instabihty, 
uniqueness, and value conflict. 

Application 1 : The ways of knowing and learning should be deeply connected with a 
practitioner’s work. Assessment of student performance in a discipline should 
reflect the mode of inquiry of the discipline involved. It also affirms embedding 
assessment in the curricula rather than adding it on. 

Principle 2 : “Through reflection, he [the practitioner] can surface and criticize the 
understandings that have grown up around the repetitive experiences of a specialized 
practice, and can make new sense of the situations of uncertainty or uniqueness, 
which he may allow himself to experience.”” 

Application 2 : Self-critical reflection by the teacher and assessor on data and experience 
is the key. The long-term aim is to develop faculty as skilled evaluators of student 
learning in their disciplines so that they can reflect on their curricula and instmction 
in light of observed results. 

Principled : “When a practitioner reflects in and on his practice, the possible objects of his 
reflection are as varied as the kinds of phenomena before him and the systems of 
knowing-in-practice, which he brings to them. He may reflect on the tacit norms 
and appreciations that underhe a judgment, or on the strategies and theories imphcit 
in a pattern of behavior. He may reflect on the feeling for a situation which has led 
him to adopt a particular course of action, on the way in which he has framed the 
problem he is trying to solve, or on the role he has constructed for himself within a 
larger institutional context. 

A pplication d : Faculty members ’ reflections must be rooted in their unique circumstances. 
Experience, not transferred information and technique, develops a feel for advanced 
achievement in a discipline. Tacit knowledge may begin with inforaiation but matures 
only with experience. In other words, faculty members should consider the ‘ ‘facts” 
of test scores and student ratings of their teaching but must also bring their intuitive 
judgment to bear on those facts and other evidence not empirically documented. 

Principle 4 : “When someone reflects-in-action, he becomes a researcher in the practice 
context. He is not dependent on the categories of estabhshed theory and technique, 
but constructs a new theory of the unique case. His inquiry is not limited to a 
deliberation about means, which depends on a prior agreement about ends. He 
does not keep means and ends separate, but defines them 



”Schon, op. cit., 49. 
^^ibid., 61. 

^ibid., 62. 
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interactively as he frames a problematic situation. He does not separate thinking from 
doing, ratiocinating his way to a decision, which he must later convert to action. 
Because his experimenting is a kind of action, implementation is built into his inquiry. 
Thus reflection-in-action can proceed, even in situations of uncertainty or 
uniqueness, because it is not bound by the dichotomies of Technical Rationahty.”*' 

Application 4 : Assessment is often mechanical and linear when it should be fluid and 
concurrent. The most advanced scientists report they were mentored by individuals 
rather than trained by transferable techniques. Soon after beginning in any field, 
the training- wheel techniques come off If this is the case for exact sciences, it is 
even more so for assessing student learning and institutional effectiveness and 
developing and implementing improvement strategies. 

As McClay observes about historians, faculty as reflective practitioners trying to improve student 
learning and academic processes will not disdain but imitate many habits of science, e.g. “the 
fastidious gathering and sifting of evidence, the efforts to be dispassionate and even-handed, the 
openness to alternative hypotheses and explanations, the caution in propoimding sweeping 
generalizations.” He goes on to say, “Although we (historians) continue to draw upon history’s 
traditional storytelling methods, we can also use sophisticated analytical models to discover patterns 
and regularities in individual and collective behavior.”^ 

As suggested before, assessment strategies and instruments should reflect a discipline’s mode of 
inquiry if they are to support faculty as reflective practitioners. A literature teacher experiences a 
student’s learning in a different medium from a music teacher. Laboratory science students obviously 
display their learning differently from sociology students. Different disciplines have preferred and 
functionally different modes of inquiry. The more congment a faculty member’s method of inquiring 
about students’ learning and the instmctor’s teaching is with the discipline’s mode of inquiry, the 
more the faculty member is likely to learn finm reflecting on assessment of student learning. 

New Techniques? No. 

We anticipate being challenged to provide methodological alternatives, e.g. “If you don’t like what 
we are doing, suggest better techniques.” We reiterate that we are not suggesting jettisoning empirical 
data. There are many substantive practices, such as the historic British external examiner system 
based on multiple reader judgment, that we should consider. Some may assume that we are making 
a case for greater use of qualitative, generic, non-discipline-based assessments; but we are not. 
Instead, we believe there are historic assessment practices, such as the British system of external 
readers, as well as jury appraisals of creative, artistic performances and products, from which we 
could learn and adapt much. 







D 



68-69. 

Scott, op. cit., 24. 
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Rather than trying to come up with new techniques to appear more scientific, we should become 
less formulaic. We seem to come on stage as technicians with a bag of assessment instruments and 
strategies and QI tools promoting our particular alchemy as the academy’s cure. Think of how this 
comes off to someone with Baizun’s perspective as illustrated by his comments on mechanization. 
He quotes Emerson: “Things are in the saddle and ride mankind.”" Then he observes: 

Emerson saw mechanism governing the mind after machineiy had coerced the body. 

The machine is an agent of ABSTRACTION. It is itself an abstraction in that it does one 
particular task (or at most two or three) and yields identical products. There is no 
fi'inge or fancy, no happy error or sudden iimovation as in the handworker’s 
performance. 

When the domestic or public landscape is filled with objects deprived of any aura, it is as 
if the world of hving things had been reduced by abstraction to something emphatically 
not alive. 

[T]he oppression of mechanism begins when every horizon is crowded with the means 
that abstract fi'om life and reduce it to fimctions." 

Communities of Judgment? Yes. 

The discernment needed to improve student learning does not lie primarily in finding ever more 
clever ways to measure students as deter min ed organisms. Instead, the more productive approach 
lies in encouraging faculties to form communities of judgment to use the hard data and intuitive 
knowledge now available to them. Scott points out that metis is not democratically distributed, 
and access to it is usually restricted to communities of practice. It only flourishes, he believes, when 
social conditions provide “a community of interest, accumulated information, and ongoing 
experimentation.”" 

Yet democratic societies seem to abhor communities of professional judgment. To avoid the 
arrogance and self interest associated with communities of professional judgment, technocratic 
democracies seek numbers rather than opinions. Porter observes, “A highly disciplined discourse 
helps to produce knowledge independent of the particular people who make it.” As he had already 
made clear, “quantification is a technology of distance,” and mathematics as a language is “highly 
structured and rule bound.”" This helps to explain the intense interest in assessing all levels of 
education through test scores in the U.S. Porter, citing Harold Lasswell, compares the U.S. and 
British pohtical dependence on numbers as follows: 

The American political system. . .made greater use of quantified, objective knowledge 
precisely because of its democratic character. By contrast, the British could rely on less 
formal modes of reasoning and communication because their political and administrative 
leaders made up a cohesive elite." 







6 



" Barzun, op. cit., 554. 
^ibid: 554-555. 
"Scott, op. cit., 334. 
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The audit mentality is now deeply rooted in “advanced” economies. They are intended to produce 
objective data, distanced from the organization while also promoting improvement. Power, at the 
London School of Economics and Political Science, comments on the distance of auditors from 
their objects of study and the consequences of that distance. 

The auditor lives in a world that the system of auditing has created and which is increasingly 
self-referential, cut off from the dimensions of organisational life for which the audit was 
designed in the first place. Such audits produce symbols of comfort for regulators and 
exist as a “ritual of inspection” which is not integrated into other systems of action: they 
terminate rather than stimulate further inquiry.** 

Porter points out the historic reciprocal relationship of individualism and rationalism. If asked to 
picture rationalism at work, most of us envision a scientist working alone in a laboratory or a lonely 
scholar in library stacks. He says even philosophers “have not known quite how to embrace a 
social conception of rationality.”*’ 

But can trust in numbers replace trust in people? Porter says, “No.” 

In short, it requires institutional or personal credibility even to produce impersonal 
numbers. . . .Trust is inseparable from objectivity, rather like a Doppelganger. But the form 
of trust supporting objectivity is anonymous and institutional rather than personal and face- 
to-face.™ 

Can we then make our way with numbers alone? Porter, like Deming, says, “Numbers alone never 
provide enough information to make detailed decisions.”” And improvement requires decisions 
based on intimate understanding. In the seventies, I (John Harris) led a relatively large instructional 
improvement effort at a large university. While this effort employed experts with advanced training 
in instructional design and technology to work with well-credentialed faculty, I noticed many 
institutions with less technical skill and resources sustained innovative pedagogies more effectively, 
and concluded that intelligence and resources without community cannot secure gains. The failure 
of the Soviet experiment indicates large, centrally plaimed management and assessment systems 
cannot deal with the complexities of human behavior. In the last analysis, sustainable improvements 
are birthed and sustained by nurturing communities. 

Conclusion 

We conclude with four points about assessment and quality improvement in higher education. 

First, we should not ask our colleagues to adopt assessment instruments and strategies, and quality 
improvement tools and processes that mechanize or technologize them or the 

** Theodore M. Porter, Trust In Numbers (Newhaven, CT: Princeton University Press, 1995), ix. 
ibid., 76. 

** Power, op. cit, 5. 

*’ Porter, op. cit., 217. 

^ibid.,2U. 

” ibid., 45 
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environments. If we truly believe in diversity, we have to include those who see persons as unique, 
relational, and free. 

Ultimately. . .a particular being is “itself’ — ^and not another one — because of its uniqueness 
which is established in communion and which renders a particular being unrepeatable as 
it forms part of a relational existence in which it is indispensable and irreplaceable.’^ 

Many academicians, for the purposes of disciplinary research, adopt an objective, impersonal, 
deterministic view of humankind. Nevertheless, there are very few, we believe, who will accept 
such assessments of their own teaching and their students’ learning, whether they are postmodern 
existentiahsts. Enlightenment humanists, rehgionists, or even empirical scientists. 

Second, we sometimes use instruments beyond their designed intent. Ewell cites Kaplan’s “law of 
the instrument” as one of assessment’s difficulties: “[T]he tendency for particular techniques, once 
developed, to be applied beyond their original intent and consequently beyond the domains in 
which they remain valid and reliable.”” Aptitude tests designed to predict performance become 
surrogate achievement measures. Personality surveys designed to allow persons within the security 
of a helping relationship to inventory themselves are used to classify students. 

Third, as has been suggested, we should begin to work with our colleagues in designing assessments 
that reflect the modes of inquiry most congruent with their respective disciplines. As Hope’s 
citations suggest, assessment of student learning, particularly at advanced levels, cannot be 
done without knowing the discipline and its lore. On the other hand, those who understand 
measurement theory can help them with content and predictive validity and particularly reliability. 
Teacher-made tests are comparatively unreliable, and as we know, validity depends on reliability. 
We have found faculty open to learning more about designing tests. 

Fourth, we could do much to improve assessment in American undergraduate education by greater 
use of second readers of student work. Claude Rawson, Yale’s Maynard Mack Professor of English, 
a graduate of Oxford and former professor at Warwick, believes the absence of a second reader in 
American undergraduate education is a major problem.” He states: 

On two-reader assessment, it is not only important but essential to ensure integrity in a 
grading system that is otherwise at the mercy of every whim, personal or pohtical prejudice, 
ambition, desire to be liked, vindictiveness, or any number of unconscious biases in any of 
us. When you add the extreme relativism that exists in the study of the humanities, and the 
inexperience of many junior instmctors whose interest is to show high grades, popularity 
with students and high course enrollments, you have a perfect recipe for endemic intellectual 
and academic laxity across a whole layer of the system, a kind of soft underbelly.” 



” J. D. Zizioulas, “Human Capacity and Human Incapacity: A Theological Exploration of Personhood” The 
Scottish Journal of Theology (1975, vol. 28), 410. 

” Ewell, op. cit. 
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Few would have the understanding of both the American and British higher education systems as 
Sheldon Rothblatt. He is internationally known for his work in the history of British higher education, 
and he is intimately acquainted with American higher education, having served as professor and 
chair of history at the University of California at Berkley and director of the Center for Studies in 
Higher Education, also at Berkley. He comments on the difficulties of adopting second readers or 
external examiners in the American system: 

I agree with Claude Rawson as to some of the benefits of the second, or in some cases 
external, examiner. But one must remember its provenance. It grew up in Britain as part of 
the single-subject honors degree. The numbers are relatively small. Students sat for 
examinations at the end of three years of study. Numbers have since grown, and the 
burdens have become great, so that pressures to cripple the British system have also 
grown. However, the real threat to the system has been from the introduction of American- 
style credit-unit courses. A typical American student course load may be as many as 10 
courses per year, or 40 courses for a B.A., so that would be 40 times that a second 
examiner would have to be foimd as opposed to the one examiner in the British system.’* 

Rothblatt is correct about the difficulty of adding second readers or external examiners to the 
current American system of 40 modular courses for a baccalaureate. There is even the further 
problem of many courses having no comprehensive final examination. The current American curricular 
system clearly works against holding students responsible for large blocks of integrated knowledge 
and skills either in general education or the major. The typical American student accumulates an 
amalgam of course credits awarded by teachers who, in most cases, individually determine the 
content and achievement expectations in their own courses. Student records clerks check transfer 
credits against brief catalog descriptions, and curriculum committees strive to preserve intellectual 
coherency while dealing with academic territorial politics. Does completing 40 courses add up to 
integrated proficiency in general education and the major? In most cases, colleges and universities 
have no way of knowing. For example, senior theses are no longer common. Nevertheless, there 
are valiant efforts to move against the tide, such as at Alvemo College, where all students are 
responsible for fulfilling eight general proficiencies. 

Despite the difficulties, we believe substantive improvement in student learning waits on faculty and 
academic administrators reflecting on practice within communities of judgment. And communities 
of judgment will require multiple assessments of substantial amounts of student work. To this point, 
the American academy appears to have decided to leave the more comprehensive assessment of 
student attainment to external, nationally-standardized tests. This seems more tolerable than dealing 
with the structural problems that mitigate 



Claude Rawson, Maynard Mack Professor of English, Yale University, personal communication, August 

16,2000. 

Rawson, op. cit. 

Sheldon Rothblatt, Professor of History Emeritus, University of California, Berkley, personal communication, Feb. 9, 
2001 . For a thorough review of the British examining system, see his book. The Modern Univer.<!ity and Iti Discontents 
(Cambridge: Cambridge University Press, 1997). 
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against someone other than the student’s teacher being the sole judge of student learning. While 
such scores can detect more global problems, they can rarely provide the detailed understanding 
needed for improvement. In this paper, we aim to describe the difficulties of relying on objectivist 
da ta alone to guide improvement. We believe there are alternative, viable curricular structures that 
will facilitate faculties functioning as communities of judgment and reflection on assessments of 
second readers and external examiners. This is a long story in itself best left to another paper. 

Summary Suggestions 

To contribute to improving higher education, we believe the quality-assessment efibrt should: 

• broaden its operant epistemology, 

• become more tentative in reporting quantitative data, 

• accept that reality cannot be completely communicated in any language, even mathematics, 

• respect the role of tacit knowledge, 

• encourage discipline-based assessment, 

• consider adopting the reflective practitioner as the dominant paradigm for higher education 
improvement, and 

• despite the difficulties encourage and support faculty as communities of professional judgment. 
Please send comments to: 



John Harris, Orlean Bullard Beeson Professor Dennis Sansom 
of Education and Associate Provost for Professor and Chair 



Quahty Assessment 
Samford University 
Birmingham, AL 35229-7020 
(205) 726-2674 
jwharris@samford.edu 



Department of Philosophy 
Samford University 
Birmingham, AL 35229-7020 
(205) 726-2839 
dlsansom@samford.edu 
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Afterword 



This paper started with my unease about the actual value that assessment and quality improvement 
contribute to improvement in higher education. Meaningful improvement begins with a deep 
understanding of its object. Assessment’s key role is to increase our understanding of student 
learning and how colleges and universities facilitate learning. It seems to me that assessment as 
now practiced does not make its optimal contribution because it is too narrowly constmcted on 
empirical objectivism to yield the depth of understanding needed for improvement. Deeper 
understandings arise from discernment and judgment of seasoned practitioners or professionals. 

After considerable experience in assessment and quality improvement, I beheve assessment will 
contribute more to improvement if it adds professional judgment to objectivist knowledge. When 
asked to comment on an early draft of this paper, Fisher Humphreys, a Samford colleague, asked: 

Why could not we employ a dialectical method for grasping tmth in which we attempt to 
keep mentored, community-dependent, judgment-achieved understanding in an ongoing 
conversation with individual, original, empirical and rational knowledge? Why should we 
make the case for judgment and understanding at the expense of objectivity and data, and 
vice-versa? Why not retain both in a continuing conversation, in the hope that what we 
need to know is more likely to be found in their conversation than by either one in isola- 
tion?' 



I agree. Nevertheless others who also reviewed earlier drafts believed that I wanted to replace 
quantitative data with professional judgment. Nothing could be further from my intention. As 
Samford’s Associate Provost for Quality Assessment, I do all that I can to promote “fact-based 
decision making” with the best numbers we can assemble. As a member of the AALE Board of 
Directors, I have facilitated the development of Viabihty Indicators, a set of enrollment and financial 
ratios to gauge the stability of colleges.* At Samford, with my encouragement and support, Jim 
Eck, director of Institutional Research, has refined a set of Key Performance Indicators [KPIs] 
that serve as dashboard indicators of quality and satisfaction. Each year, he presents these Viability 
Indicators and KPIs to the president and his leadership team. We collect and use myriad other 
statistics on all aspects of the University, with particular attention to measures of student learning. 

Valid and accurate quantifiable data are indispensable in evaluating academic programs and 
processes. At the same time, I believe the assessment and quality improvement movements with 
which I have been identified for all of my career fail to optimize their contribution by trying so hard 
to be a science by their excessive preoccupation with quantifiable data. 



*Thomas E. Corts, Samford president, and James Eck, Samford director of Institutional Research, summarize 
this work in “Seaworthiness: Ten critical measures” in a forth coming issue of Trusteeship, Washington: 
Association of Governing Boards, vol. 9, no. 3 (May/June) 

‘ Fisher H. Humphreys, Professor, Beeson Divinity School, Samford University, July 2000. 
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Oakeshott’s summary of Pascal’s concern about rationalism’s hegemony reflects my concern about 
higher education assessment’s exclusion of professional judgment: 

[F]ew detected more surely than Pascal that the significance of Rationalism is not its 
recognition of technical knowledge, but its failure to recognize any other; its philosophical 
error lies in the certainty it attributes to technique and in its doctrine of the sovereignty of 
technique...^ 

I hope this paper in some way might lead assessment and quality improvement experts to rethink 
their epistemological presuppositions so that they can do as Humphreys suggests: 

[EJmploy a dialectical method for grasping truth in which we attempt to keep 
mentored, community-dependent, judgment-achieved understanding in an ongoing 
conversation with individual, original, empirical and rational knowledge.^ 

In summarizing this paper, I offer the following sequential statements, intentionally spare and stark: 

1 . Improved student learning is the aim of assessment and quality improvement. 

2 . Student learning is significantly related to curricula, methods of instruction, and institutional 
culture and practices. 

3 . Tme improvement depends on increased understanding. 

4. Assessment’s value is determined by its contribution to increasing understanding. 

5 . Objectivist, quantitative data contribute to increased understanding by alerting institutions 
to performance problems and in analyzing causes, e.g. a large percentage of professional 
graduates failing licensing examinations or a low percentage of students graduating in four 
to six years. 

6 . Measurement aims for precision that increases with simplification. 

7 . Simplification brackets many important variables in human organizations that cannot be 
operationally defined, measured, or even identified. 

8 . Inevitably, subjective judgment must interpret objectivist information and take into account 
the unmeasured and immeasurables. 

9. Subjective judgment, tacit knowledge or metis, comes from intimate and sustained 
involvement with a practice. 

1 0 . Subj ective judgment may be validated by intersubj ective agreement. 

1 1 . Professional practitioners develop and nurture intersubjective agreement through community. 

1 2 . Assessment contributes most to improvement of student learning through communities of 
judgment. 



John Harris 



^ Michael Oakeshott, Rationalism In Politics (New York: Barnes & Noble Books, 1 962), 2 1 . 
^Humphrey, op. cit. 
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